{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
    "import pandas as pd\n",
    "import random\n",
    "import checklist\n",
    "from checklist.editor import Editor\n",
    "from checklist.expect import Expect\n",
    "from checklist.pred_wrapper import PredictorWrapper\n",
    "from checklist.test_types import MFT\n",
    "from typing import List\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f075dfff110>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize random seed\n",
    "# Remove this code to experiment with random samples\n",
    "random.seed(123)\n",
    "torch.manual_seed(456)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MFTs: Introduction\n",
    "In this notebook, we will create Minimum Functionality Tests (MFTs) for a generative language model. MFTs test one specific function of a language model. They are analogous to unit tests in traditional software engineering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup generative model\n",
    "Before we can test anything, we need to set up our language model. We will use the HuggingFace transformers library to load a GPT2 model.\n",
    "\n",
    "First, we create a tokenizer. The tokenizer is responsible for splitting strings into individual words, then converting those words into vectors of numbers that our model can understand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8496, 754, 1242, 14210, 43989, 30]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load pretrained model tokenizer\n",
    "tokenizer = GPT2Tokenizer.from_pretrained('gpt2')\n",
    "\n",
    "# Demonstrate what the tokenizer does\n",
    "tokenizer.encode(\"Wherefore art thou Romeo?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our tokenizer has turned the human-readable text into a list of numbers that the model understands. Next, let's load the GPT2 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model loaded'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load pretrained model (weights)\n",
    "model = GPT2LMHeadModel.from_pretrained(\"gpt2\", pad_token_id=tokenizer.eos_token_id)\n",
    "device = 'cuda'\n",
    "model.eval()\n",
    "model.to(device)\n",
    "\"Model loaded\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating text with the model requires a bit of work. Let's write a function `generate_sentences` to handle the text generation.\n",
    "\n",
    "`generate_sentences` has 1 parameter, `prompts`, which is a list of strings. A prompt is a string that the model will use as a starting point for generating new text. It gives the model context about what kind of text should be generated.\n",
    "\n",
    "`generate_sentences` will output a list of generated text responses for each prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sentences(prompts: List[str]) -> List[str]:\n",
    "    sentences = []\n",
    "    for prompt in prompts:\n",
    "        token_tensor = tokenizer.encode(prompt, return_tensors='pt').to(device) # return_tensors = \"pt\" returns a PyTorch tensor\n",
    "        out = model.generate(\n",
    "            token_tensor,\n",
    "            do_sample=True,\n",
    "            min_length=10,\n",
    "            max_length=50,\n",
    "            num_beams=1,\n",
    "            temperature=1.0,\n",
    "            no_repeat_ngram_size=2,\n",
    "            early_stopping=False,\n",
    "            output_scores=True,\n",
    "            return_dict_in_generate=True)\n",
    "        text = tokenizer.decode(out.sequences[0], skip_special_tokens=True)\n",
    "        sentences.append(text[len(prompt):])\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' if it were not, then thou art a devil, with his body made of the fire: thy body he that is not with fire makes of it. But thou, however, have done evil and have been condemned.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sentences([\"Wherefore art thou Romeo?\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that everything is ready, we can write our first MFT.\n",
    "\n",
    "## MFT - Language prompt\n",
    "For this MFT, we will expect the model to create a reasonable continuation of a prompt. The model will be prompted with strings like \"The most commonly spoken language in {country} is \" where {country} is a placeholder for a country such as Spain.\n",
    "\n",
    "We need create a rule to determine if the model passes our test. The criteria for passing or failing the test is entirely user defined. We will consider this MFT to pass if the model's output contains any language name. This will demonstrate that the model understands the general context of the prompt. The mentioned language doesn't have to be accurate - for example, \"In Spain the most commonly spoken language is Indonesian\" would pass our test, because Indonesian is a language. The language may also be located anywhere in the output - for example, \"In Spain the most commonly spoken language is not easy to learn. Spanish has many complicated conjugations.\" would also pass our test.\n",
    "\n",
    "In a later section of this notebook, there is another version of this MFT that is stricter, requiring the correct language to be mentioned in the response."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handwritten MFT\n",
    "First, we will write the MFT by hand. Then, we'll use Checklist's MFT class to demonstrate how Checklist helps us create the MFT much more quickly.\n",
    "\n",
    "#### Generate prompts from template\n",
    "We will use Checklist's Editor class to quickly create the prompts. For a detailed explanation of generating data, see the \"1. Generating data\" tutorial notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The most commonly spoken language in United States is',\n",
       " 'The most commonly spoken language in France is',\n",
       " 'The most commonly spoken language in Guatemala is',\n",
       " 'The most commonly spoken language in Mongolia is',\n",
       " 'The most commonly spoken language in Japan is']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "editor = Editor()\n",
    "# Note: remove the country parameter to generate prompts with random countries\n",
    "prompt_strs = editor.template(\"The most commonly spoken language in {country} is\", country = [\"United States\", \"France\", \"Guatemala\", \"Mongolia\", \"Japan\"])\n",
    "prompt_strs.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Language CSV\n",
    "We need a list of languages to check if the model's output contains a language. To save some time, we will read language names from a CSV file. The data comes from standard ISO Language Codes https://datahub.io/core/language-codes "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha2</th>\n",
       "      <th>English</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa</td>\n",
       "      <td>Afar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ab</td>\n",
       "      <td>Abkhazian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ae</td>\n",
       "      <td>Avestan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>af</td>\n",
       "      <td>Afrikaans</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ak</td>\n",
       "      <td>Akan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>yi</td>\n",
       "      <td>Yiddish</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>yo</td>\n",
       "      <td>Yoruba</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>za</td>\n",
       "      <td>Zhuang; Chuang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>zh</td>\n",
       "      <td>Chinese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>zu</td>\n",
       "      <td>Zulu</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>184 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    alpha2         English\n",
       "0       aa            Afar\n",
       "1       ab       Abkhazian\n",
       "2       ae         Avestan\n",
       "3       af       Afrikaans\n",
       "4       ak            Akan\n",
       "..     ...             ...\n",
       "179     yi         Yiddish\n",
       "180     yo          Yoruba\n",
       "181     za  Zhuang; Chuang\n",
       "182     zh         Chinese\n",
       "183     zu            Zulu\n",
       "\n",
       "[184 rows x 2 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "urllib.request.urlretrieve('https://datahub.io/core/language-codes/r/language-codes.csv', 'language-codes.csv')\n",
    "lang_codes_csv = pd.read_csv('language-codes.csv')\n",
    "lang_codes_csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the MFT\n",
    "Now we're ready to create the MFT. We will create 3 Pandas dataframes, one each for prompts, responses, and results. Then, we will loop over the prompts, send each prompt to the model, and determine if it passes or fails the test. Each prompt and its test result will be recorded in the dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = pd.DataFrame({\"id\": [], \"prompt\": []})\n",
    "responses = pd.DataFrame({\"id\": [], \"response\": []})\n",
    "results = pd.DataFrame({\"id\": [], \"p/f\": []})\n",
    "langs = lang_codes_csv[\"English\"].tolist()\n",
    "\n",
    "model_responses = generate_sentences(prompt_strs.data)\n",
    "\n",
    "for (i, response) in enumerate(model_responses):\n",
    "    pf = 'fail'\n",
    "    \n",
    "    # Check if any language from the CSV data is in the generated string\n",
    "    for l in langs:\n",
    "        if l in response:\n",
    "            pf = 'pass'\n",
    "            break\n",
    "\n",
    "    prompts = prompts.append({\"id\": i, \"prompt\": prompt_strs.data[i]}, ignore_index=True)\n",
    "    responses = responses.append({\"id\": i, \"response\": response}, ignore_index=True)\n",
    "    results = results.append({\"id\": i, \"p/f\": pf}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show test results\n",
    "Now let's look at the results of our test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"max_colwidth\", 250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>prompt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>The most commonly spoken language in United States is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>The most commonly spoken language in France is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>The most commonly spoken language in Guatemala is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>The most commonly spoken language in Mongolia is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>The most commonly spoken language in Japan is</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                                                 prompt\n",
       "0  0.0  The most commonly spoken language in United States is\n",
       "1  1.0         The most commonly spoken language in France is\n",
       "2  2.0      The most commonly spoken language in Guatemala is\n",
       "3  3.0       The most commonly spoken language in Mongolia is\n",
       "4  4.0          The most commonly spoken language in Japan is"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>English. Nearly all U.S. adult males speak English well above grade level and in the U-1 category the average verbal IQ is 98 percent. For higher grade language learners such as children and others</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>\"l'homme\", or what one French word for \"father\", means.\\n\\nBut in the United States, which had more than 60,000 native-born people in 2010, there was nothing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>Guatemalan and has a slightly older pronunciation (14 years old).\\n\\nMost of these languages have more or less the same vocabulary, as they come from different parts of the world. Most common is English (</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>Ainu, but an interesting source for these terms is the Yuktai language spoken in the country near the border with Turkmenistan. Another official official word is Shum, and it is easy to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>\"Fuku wo kun-mitsu.\" It means \"A man cannot run.\" However, there are other ways of speaking the word—such as \"kimasu.\"\\n\\nThe Kansai</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id  \\\n",
       "0  0.0   \n",
       "1  1.0   \n",
       "2  2.0   \n",
       "3  3.0   \n",
       "4  4.0   \n",
       "\n",
       "                                                                                                                                                                                                        response  \n",
       "0          English. Nearly all U.S. adult males speak English well above grade level and in the U-1 category the average verbal IQ is 98 percent. For higher grade language learners such as children and others  \n",
       "1                                                  \"l'homme\", or what one French word for \"father\", means.\\n\\nBut in the United States, which had more than 60,000 native-born people in 2010, there was nothing  \n",
       "2   Guatemalan and has a slightly older pronunciation (14 years old).\\n\\nMost of these languages have more or less the same vocabulary, as they come from different parts of the world. Most common is English (  \n",
       "3                      Ainu, but an interesting source for these terms is the Yuktai language spoken in the country near the border with Turkmenistan. Another official official word is Shum, and it is easy to  \n",
       "4                                                                           \"Fuku wo kun-mitsu.\" It means \"A man cannot run.\" However, there are other ways of speaking the word—such as \"kimasu.\"\\n\\nThe Kansai  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>p/f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>fail</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id   p/f\n",
       "0  0.0  pass\n",
       "1  1.0  pass\n",
       "2  2.0  pass\n",
       "3  3.0  pass\n",
       "4  4.0  fail"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can merge all the dataframes to make the results easier to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>p/f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>The most commonly spoken language in United States is</td>\n",
       "      <td>English. Nearly all U.S. adult males speak English well above grade level and in the U-1 category the average verbal IQ is 98 percent. For higher grade language learners such as children and others</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>The most commonly spoken language in France is</td>\n",
       "      <td>\"l'homme\", or what one French word for \"father\", means.\\n\\nBut in the United States, which had more than 60,000 native-born people in 2010, there was nothing</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>The most commonly spoken language in Guatemala is</td>\n",
       "      <td>Guatemalan and has a slightly older pronunciation (14 years old).\\n\\nMost of these languages have more or less the same vocabulary, as they come from different parts of the world. Most common is English (</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>The most commonly spoken language in Mongolia is</td>\n",
       "      <td>Ainu, but an interesting source for these terms is the Yuktai language spoken in the country near the border with Turkmenistan. Another official official word is Shum, and it is easy to</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>The most commonly spoken language in Japan is</td>\n",
       "      <td>\"Fuku wo kun-mitsu.\" It means \"A man cannot run.\" However, there are other ways of speaking the word—such as \"kimasu.\"\\n\\nThe Kansai</td>\n",
       "      <td>fail</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                                                 prompt  \\\n",
       "0  0.0  The most commonly spoken language in United States is   \n",
       "1  1.0         The most commonly spoken language in France is   \n",
       "2  2.0      The most commonly spoken language in Guatemala is   \n",
       "3  3.0       The most commonly spoken language in Mongolia is   \n",
       "4  4.0          The most commonly spoken language in Japan is   \n",
       "\n",
       "                                                                                                                                                                                                        response  \\\n",
       "0          English. Nearly all U.S. adult males speak English well above grade level and in the U-1 category the average verbal IQ is 98 percent. For higher grade language learners such as children and others   \n",
       "1                                                  \"l'homme\", or what one French word for \"father\", means.\\n\\nBut in the United States, which had more than 60,000 native-born people in 2010, there was nothing   \n",
       "2   Guatemalan and has a slightly older pronunciation (14 years old).\\n\\nMost of these languages have more or less the same vocabulary, as they come from different parts of the world. Most common is English (   \n",
       "3                      Ainu, but an interesting source for these terms is the Yuktai language spoken in the country near the border with Turkmenistan. Another official official word is Shum, and it is easy to   \n",
       "4                                                                           \"Fuku wo kun-mitsu.\" It means \"A man cannot run.\" However, there are other ways of speaking the word—such as \"kimasu.\"\\n\\nThe Kansai   \n",
       "\n",
       "    p/f  \n",
       "0  pass  \n",
       "1  pass  \n",
       "2  pass  \n",
       "3  pass  \n",
       "4  fail  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged = pd.merge(responses, results, on=\"id\")\n",
    "merged = pd.merge(prompts, merged, on=\"id\")\n",
    "merged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's display the failing tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>p/f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>The most commonly spoken language in Japan is</td>\n",
       "      <td>\"Fuku wo kun-mitsu.\" It means \"A man cannot run.\" However, there are other ways of speaking the word—such as \"kimasu.\"\\n\\nThe Kansai</td>\n",
       "      <td>fail</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                                         prompt  \\\n",
       "4  4.0  The most commonly spoken language in Japan is   \n",
       "\n",
       "                                                                                                                                response  \\\n",
       "4   \"Fuku wo kun-mitsu.\" It means \"A man cannot run.\" However, there are other ways of speaking the word—such as \"kimasu.\"\\n\\nThe Kansai   \n",
       "\n",
       "    p/f  \n",
       "4  fail  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged.loc[merged['p/f'] == 'fail']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test with Checklist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's try running the MFT with Checklist. We will no longer need to keep track of results in Pandas dataframes, since Checklist will track the results for us.\n",
    "\n",
    "#### Create the expectation function\n",
    "In order to determine if an example passes or fails the test, Checklist uses an expectation function. An expectation function is a function that receives the example, then returns true if the example passes the test, or false if the example fails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def response_contains_language(x, pred, conf, label=None, meta=None):\n",
    "    for l in langs:\n",
    "        if l in pred:\n",
    "            return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will wrap this function with `Expect.single`, which causes the expectation function to be called for each example. In other cases, you might want to have an expectation function that checks multiple examples simulatneously. See the tutorial notebook \"3. Test types, expectation functions, running tests\" for detailed information about expectation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "contains_language_expect_fn = Expect.single(response_contains_language)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can feed our prompts and expectation function into the MFT constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = MFT(**prompt_strs, name='Language in response', description='The response contains a language.', expect=contains_language_expect_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to run the test, Checklist also needs a function that generates the model's predictions for the inputs. The function receives all inputs (prompts) as a list, and must return the results in a tuple `(model_predictions, confidences)`, where `model_predictions` is a list of all the predictions, and `confidences` is a list of the model's scores for those predictions.\n",
    "\n",
    "We will not be using confidences in this test. Checklist provides a wrapper function `PredictorWrapper.wrap_predict()` that outputs a tuple with a confidence score of 1 for any prediction. We can use it to wrap `generate_sentences` so the predictions will have a confidence score as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['русского.\\n\\nThe Portuguese word for \"pursuit\" is a sort of \"suit,\" also in the form уми, a kind of shell'],\n",
       " array([1.]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wrapped_generator = PredictorWrapper.wrap_predict(generate_sentences)\n",
    "wrapped_generator([\"The most commonly spoken language in Brazil is \"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready to run the test. The first argument to the `test.run()` function is the generator function we just created. We will also set the optional parameter `overwrite=True` so the test can be re-run without an error. If overwrite=False, then Checklist will reject subsequent test runs to prevent us from accidentally overwriting your test results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting 5 examples\n"
     ]
    }
   ],
   "source": [
    "test.run(wrapped_generator, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the results, we can use the `summary` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_example(x, pred, conf, label=None, meta=None): \n",
    "    return 'Prompt:      %s\\nCompletion:      %s' % (x, pred) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test cases:      5\n",
      "Fails (rate):    1 (20.0%)\n",
      "\n",
      "Example fails:\n",
      "Prompt:      The most commonly spoken language in Guatemala is\n",
      "Completion:       Guatemalan.\n",
      "\n",
      "Gabor Banda was born in Buntamas the day he was arrested. The elder Babor (35) came to Guatemala in 1892 but left after the war, and stayed\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "test.summary(format_example_fn = format_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test results can also be explored visually by using the `visual_summary` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30db79d879334738aece2c67f28ba862",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "TestSummarizer(stats={'npassed': 4, 'nfailed': 1, 'nfiltered': 0}, summarizer={'name': 'Language in response',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test.visual_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MFT - Language prompt with accurate response\n",
    "\n",
    "Let's make our test a little stricter to better understand the model's behavior. We will now require the model to respond with the correct language instead of any language in general. By using the `meta=True` argument for `editor.template()`, the country associated with the prompt will be will be stored in the `country_prompts` object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "country_prompts = editor.template(\"The most commonly spoken language in {country} is  \", country = [\"United States\", \"France\", \"Guatemala\", \"Mongolia\", \"Japan\"], meta=True)\n",
    "correct_responses = {\n",
    "    \"United States\": \"English\",\n",
    "    \"France\": \"French\",\n",
    "    \"Guatemala\": \"Spanish\",\n",
    "    \"Mongolia\": \"Mongolian\",\n",
    "    \"Japan\": \"Japanese\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The country metadata can be accessed with `country_prompts.meta`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'country': 'United States'},\n",
       " {'country': 'France'},\n",
       " {'country': 'Guatemala'},\n",
       " {'country': 'Mongolia'},\n",
       " {'country': 'Japan'}]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_prompts.meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handwritten Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = pd.DataFrame({\"id\": [], \"prompt\": []})\n",
    "responses = pd.DataFrame({\"id\": [], \"response\": []})\n",
    "test_results = pd.DataFrame({\"id\": [], \"p/f\": []})\n",
    "\n",
    "model_responses = generate_sentences(country_prompts.data)\n",
    "\n",
    "for (i, response) in enumerate(model_responses):\n",
    "    pf = 'fail'\n",
    "    country = country_prompts.meta[i][\"country\"]\n",
    "    \n",
    "    # Check if the correct language is in the response\n",
    "    language = correct_responses[country]\n",
    "    if language in response:\n",
    "        pf = 'pass'\n",
    "\n",
    "    prompts = prompts.append({\"id\": i, \"prompt\": country_prompts.data[i]}, ignore_index=True)\n",
    "    responses = responses.append({\"id\": i, \"response\": response}, ignore_index=True)\n",
    "    test_results = test_results.append({\"id\": i, \"p/f\": pf}, ignore_index=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show test results\n",
    "Let's look at our test results. The first dataframe contains the prompts given to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>prompt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>The most commonly spoken language in United States is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>The most commonly spoken language in France is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>The most commonly spoken language in Guatemala is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>The most commonly spoken language in Mongolia is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>The most commonly spoken language in Japan is</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                                                   prompt\n",
       "0  0.0  The most commonly spoken language in United States is  \n",
       "1  1.0         The most commonly spoken language in France is  \n",
       "2  2.0      The most commonly spoken language in Guatemala is  \n",
       "3  3.0       The most commonly spoken language in Mongolia is  \n",
       "4  4.0          The most commonly spoken language in Japan is  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next dataframe shows the model's response to the prompt (not including the prompt itself)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>ˈeˌs, which is pronounced by its singular, plural form. While both ī and ē can sometimes be pronounced English, ɒ is generally not. Instead, I</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>ˈmʒbɛt (to translate, speak, etc.). The verb \"to say\" stands for in English, which has become the first Latin language where the meaning and pronunciation is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>əʒ (i.e., yam) which means \"one who sits up and speaks as a yahtzee.\" Yaht is spoken by two groups of individuals: the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>̈̄̇̅ (the Mongolian language), which in many other parts also means I (of). Mongolians also call the English spoken by the Greeks \"gok\", while</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>ō. The noun is often translated as \"unpleasant\", \"sore\", and \"disgusting\". It's an English word that makes sense if you consider it as an American noun</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id  \\\n",
       "0  0.0   \n",
       "1  1.0   \n",
       "2  2.0   \n",
       "3  3.0   \n",
       "4  4.0   \n",
       "\n",
       "                                                                                                                                                        response  \n",
       "0                 ˈeˌs, which is pronounced by its singular, plural form. While both ī and ē can sometimes be pronounced English, ɒ is generally not. Instead, I  \n",
       "1  ˈmʒbɛt (to translate, speak, etc.). The verb \"to say\" stands for in English, which has become the first Latin language where the meaning and pronunciation is  \n",
       "2                                         əʒ (i.e., yam) which means \"one who sits up and speaks as a yahtzee.\" Yaht is spoken by two groups of individuals: the  \n",
       "3                 ̈̄̇̅ (the Mongolian language), which in many other parts also means I (of). Mongolians also call the English spoken by the Greeks \"gok\", while  \n",
       "4        ō. The noun is often translated as \"unpleasant\", \"sore\", and \"disgusting\". It's an English word that makes sense if you consider it as an American noun  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "responses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final dataframe shows the pass/fail status of the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>p/f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>fail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>fail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>fail</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id   p/f\n",
       "0  0.0  pass\n",
       "1  1.0  fail\n",
       "2  2.0  fail\n",
       "3  3.0  pass\n",
       "4  4.0  fail"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with Checklist\n",
    "Now let's run the test with Checklist. All we need is a new expectation function. The rest of the process is the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def response_contains_correct_language(x, pred, conf, label=None, meta=None):\n",
    "    country = meta['country']\n",
    "    language = correct_responses[country]\n",
    "    return language in pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_language_expect_fn = Expect.single(response_contains_correct_language)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = MFT(**country_prompts, name='Correct language in response', description='The response contains the correct language for the country in the prompt.', expect=correct_language_expect_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting 5 examples\n"
     ]
    }
   ],
   "source": [
    "test.run(wrapped_generator, overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test cases:      5\n",
      "Fails (rate):    2 (40.0%)\n",
      "\n",
      "Example fails:\n",
      "Prompt:      The most commonly spoken language in Mongolia is  \n",
      "Completion:      ike (\"labor\") and ki (\"work\").  This is a traditional term which means for any of three things:\n",
      " ikki means the work done on one's personal work \n",
      "----\n",
      "Prompt:      The most commonly spoken language in Guatemala is  \n",
      "Completion:      ͡°  as اجث الرجة ااتلوحهى لشاس ملا حم\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "test.summary(format_example_fn = format_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6536bbee1b8a436c9829ed4e4eb96598",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "TestSummarizer(stats={'npassed': 3, 'nfailed': 2, 'nfiltered': 0}, summarizer={'name': 'Correct language in re…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test.visual_summary()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
